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Introduction 


In  the  SE  community,  it  is  important  for  systems  to  be  agile  and  rapidly  and  effectively  adapt  to  sudden  changes  in 
the  environment.  Agility  in  SE  is  found  in  two  general  areas  -  process  and  product.  Process  agility  provides  systems 
engineers  with  the  methods,  processes  and  tools  necessary  to  operate  more  effectively  in  development  environments 
driven  by  change.  The  ability  to  rapidly  adapt  is  necessary  while  working  with  an  increasing  rate  of  technology 
advancement,  an  increasing  need  for  interoperability  between  legacy  and  new  capabilities,  evolving  requirements 
throughout  the  development  lifecycle,  and  the  changing  economic  and  political  factors  that  undergird  and  enable 
system  development.  Perhaps  one  of  the  most  important  concepts  in  Agile  SE  is  the  reconciliation  and  integration  of 
systems  and  software  engineering  activities.  If  software  development  processes  are  to  operate  seamlessly  with  SE 
processes,  SE  processes  must  borrow  notions  of  agility  and  flexibility  found  in  software  engineering. 

The  purpose  of  RT-124  is  to  identify,  describe,  and  evaluate  possible  methods,  practices  or  tools  (enablers)  that 
could  improve  the  ability  of  systems  engineering  to  adapt  to  changing  development  environments.  In  order  to 
efficiently  make  use  of  scarce  research  resources,  RT-124  has  established  a  triage  process  for  identifying  and  then 
rapidly  evaluating  the  probability  of  effectiveness  of  candidate  enablers  as  they  are  identified.  The  ultimate  result  of 
the  process  is  an  evaluation  white  paper  supporting  one  of  three  decisions: 

1.  not  likely  to  be  effective, 

2.  possibly  suitable  but  more  research  is  needed,  or 

3.  definitely  suitable  and  expedited  transition  is  recommended. 

This  paper  describes  the  process  and  its  products.  After  each  execution  of  the  process,  a  reflection  activity  will  be 
held  to  identify  strengths  and  weaknesses  of  the  process  and  to  identify  and  make  appropriate  improvements. 

2  Selection  and  Evaluation  Process  Overview 


The  overall  process,  as  illustrated  in  Figure  1,  leverages  nearly  a  decade  of  research  into  practice  description, 
evaluation  and  dissemination  represented  by  the  DoD  Acquisition  Best  Practices  Clearinghouse  (BPCh)..1  [1,  2,  3]  The 
process  itself  can  operate  concurrently  for  a  number  of  enablers,  and  the  actual  cadence  can  be  adjusted  by  the 
number  of  enablers  under  consideration  and  the  number  and  availability  of  evaluators. 


2.1  Identification 

Enablers  can  be  found  in  many  environments,  disciplines,  and  activities.  Real  value  can  be  achieved  when  a 
process  used  in  one  discipline  can  be  adapted  quickly  to  provide  value  in  a  different  discipline.  RT-124  attempts  to 
identify  enablers  by  monitoring  the  agile,  lean,  and  adaptive  research  and  practice  ecosystems.  Generally,  the  most 
efficient  way  of  tapping  into  the  communities  is  via  existing  communities  of  practice.  This  can  be  achieved  through 
monitoring  communications  in  social  media  groups  and  websites  (such  as  Linkedln  or  Facebook  groups  associated 
with  the  Scaled  Agile  Framework,  Lean  Enterprise  Institute,  Agile  Alliance,  Lean  Systems  Society  and  Model-based 
Systems  Engineering),  reading  conference  proceedings,  attending  workshops,  and  participating  in  working  groups 
(such  as  the  INCOSE  Agile  Systems  Engineering  WG). 

Identification,  however  needs  to  employ  a  set  of  common  criteria  so  that  obviously  inappropriate  enablers  are  not 
pursued.  The  identification  criteria  developed  for  RT-124  are  based  on  earlier  SERC  work.  [4,  5,  6]: 


Operated  by  DAU,  the  BPCh  was  a  web-enabled  best  practice  repository  and  selection  tool  residing  within  the  DAU  knowledge  management  system 
and  associated  with  DAU's  acquisition  communities  of  practice.  The  BPCh  operated  through  2010. 
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•  Supports  some  aspect  of  agility  or  leanness  (e.g.  small  batch  size,  incremental/iterative  development,  value  to 
the  customer) 

•  Is  reasonably  defined  (there  is  a  somewhat  standard  definition) 

•  Aligns  with  at  least  one  of  the  SEBOK  systems  engineering  primary  discipline  areas 

•  Sufficient  information  exists  to  characterize  it 


/  Monitor  Enabler  Sources  ) 


Figure  1:  Overall  Process 


2.2  Characterization  and  Evaluation 

Characterization  consists  of  researching  the  identified  enabler,  gathering  any  evidence  about  its  use  and  the 
results,  if  possible  interviewing  organizations  that  have  applied  it,  and  ideally  (but  rarely),  finding  any  empirical 
studies  regarding  it. 

To  provide  for  a  common  language  (ontology)  and  to  enable  continuous  and  consistent  assimilation  of  information 
over  time,  it  is  appropriate  to  establish  attributes  to  describe  each  enabler.  The  attributes  are  organized  to  support 
the  evaluation  criteria.  Characterization  attributes  and  their  assessment  scale  are  shown  in  Table  1.  The  attributes  are 
intentionally  broad  to  support  a  fairly  rapid  assessment  of  potential.  The  evaluation  criteria  are  shown  in  Table  2. 

Evaluation  activities  are  centered  around  a  single  researcher  identifying  evidence  from  various  sources,  discussing 
the  enabler  with  experts  in  its  creation  or  use  as  well  as  with  system  engineering  practitioners  and  managers.  This 
information  is  then  reflected  in  the  attributes.  Information  from  the  attribute  evaluation  is  provided  to  the  research 
team,  including  a  statistically  based  score  for  each  criterion.  This  score  is  considered,  but  is  not  the  only  input  to  the 
decision  making  process.  In  general,  the  score  for  impact  and  relevance  take  precedence,  since  research  can  usually 
mitigate  weaknesses  in  maturity  and  adoptability.  However,  lower  scores  indicate  that  the  team  should  be  very  clear 
about  the  relationship  between  the  possible  benefit  and  the  cost  of  proceeding. 

If  the  researcher  and  the  team  believe  that  the  enabler  is  simply  not  suitable,  or  that  while  it  may  show  promise, 
the  expense  or  extent  of  additional  research  does  not  seem  to  match  the  benefit,  the  enabler  is  discarded  and  the 
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information  filed  as  notes.  If  the  team  believes  there  is  sufficient  merit  to  do  additional  research,  or  if  there  is  an 
indication  that  the  enabler  is  already  applicable,  a  white  paper  is  generated  and  delivered  to  the  sponsor. 


Table  1:  Enabler  Attributes 


Attribute 

Description 

Agile/Adaptive  Impact  Attributes  I 

Evaluation  Scale: 

Unknown  -  Not  determinable  at  this  time  (score:  <null>) 

None  -  Currently  cannot  support  this  attribute  ( score:  -1) 

Partial  Support  -  Does  not  negate  this  attribute  ( score:  0) 

Explicit  Support  -  Designed  to  support  this  attribute  ( score:  +1) 

Batch  Size 

Limiting  or  supporting  smaller  batch  sizes  for  SE  activities 

Iteration 

Supporting  iterative  development  capability 

SE  Activity  Value 

Determining  the  value  of  SE  activities  to  support  better  SE  efficiency  and  effectiveness 

Customer  Value 

Accelerating  the  delivery  of  value  to  the  customer 

Work  In  Progress 

Visibility  of  existing  WIP  or  limiting  WIP  to  increase  flow  and  protect  scarce  resources 

Scheduling 

Flexibility  to  handle  multiple  priority  tasks  without  unnecessary  perturbation  of  engineering  flow 

Requirement 

Evolution 

Changing/emergent  requirements  and  the  ability  to  evolve  systems  over  time 

Discipline 

Better/faster/more  effective  communication  and  more  rapid  integration  between  various 

integration 

disciplines  as  changes  occur 

Artifacts 

Development  of  fewer,  higher-value  artifacts  that  are  easier  to  maintain  congruent 

Stakeholder 

Management 

Effective  and  adaptive  balancing  of  stakeholder  needs 

Relevance  Attributes 

Evaluation  Scale: 

Unknown  -  Not  determinable  at  this  time  (score:  <null>) 

None  -  Currently  cannot  support  this  attribute  (score:  -1) 

Partial  Support  -  Does  not  negate  this  attribute  ( score:  0) 

Explicit  Support  -  Designed  to  support  the  attribute  ( score:  +1) 

Scalability 

Can  apply  to  all  types  of  systems  from  simple  to  ultra-large  SoSs  with  deep  supplier  chains  and 
multiple  concurrent  and  interacting  initiatives. 

Criticality 

Can  apply  where  there  are  stringent  safety,  security,  or  mission-critical  requirements 

Adaptability 

Can  adapt  or  extend  to  apply  to  different  SE  disciplines,  domains  or  development  circumstances 

Maturity  and  Repeatability  Attributes  I 

Evaluation  Scale: 

Unknown  -  Not  determinable  at  this  time  (score:  <null>) 

None  -  does  not  currently  meet  this  attribute  ( score:  -1) 

Partial  Support  -  Weakly  meets  this  attribute  ( score:  0) 

Explicit  Support  -  Strongly  meets  the  attribute]  ( score:  +1) 

Definition 

Is  defined  sufficiently  to  be  studies/replicated. 

Experience 

Is  implemented  or  used  in  multiple  instances 

Breadth 

Has  been  applied  over  a  range  of  different  types  of  organizations  or  application  areas  (e.g. 
acquirers,  developers,  integrators;  business,  communications,  defense,  medicine,  space,  cyber¬ 
physical) 

Media  Presence 

Is  meaningfully  referenced  (e.g.  reviews,  analyses,  case  studies)  directly  or  in  analogy  in  technical 
media  (e.g.  journals,  technical  reports,  respected  blogs) 
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Table  1.  Enabler  Attributes  (cont.) 


Attribute 

Description 

Adoptability  Attributes 

Evaluation  Scale: 

Unknown  -  Not  determinable  at  this  time  (score:  <null>) 

None  -  does  not  currently  meet  this  attribute  ( score:  -1) 

Partial  Support  -  Weakly  meets  this  attribute  ( score:  0) 

Explicit  Support  -  Strongly  meets  the  attribute]  ( score:  +1) 

Ease  of  Use 

Can  be  learned  and  applied  by  non-experts 

Latency 

Impacts  SE  agility  within  an  acceptable  time  frame 

Cost  to  Deploy 

Investment  costs  (e.g.,  special  equipment,  training)  to  implement  the  enabler  are  acceptable 

Cost  to  Use 

Execution  costs  (licenses,  additional  staff  time)  for  the  enabler  are  acceptable 

Table  2:  Evaluation  Criteria 


Criteria 

Description 

Impact 

High  impact  in  at  least  one  agile  attribute  and  some  impact  in  more  than  one  additional  area 

Relevance 

And 

Maturity  and 
Repeatablility 

Sufficiently  well  defined  that  implementation  is  portable  to  other  projects;  Used  successfully  in  at 
least  one  SE-like  context. 

Adoptability 

Are  sufficiently  related  to  the  culture  and  processes  of  current  systems  engineering  practice  so  as  not 
to  be  rejected  by  the  majority  of  the  workforce;  do  not  require  overly  burdensome  restructuring  of 
organizational  governance  or  statutory  changes 

2.3  Development  of  the  evaluation  white  paper 

Each  white  paper  will  provide  the  following  information: 

Summary  of  Evaluation  Assessment  and  Recommendations  for  the  Enabler 
Part  I:  Description  of  the  Enabler. 

A  description  of  the  enabler  including  any  pertinent  information  as  to  its  source,  its  use,  and  its 
relationship  to  other  enablers  or  existing  processes.  This  section  may  be  very  short  or  significant 
depending  on  the  recommendations 

Part  II:  Evaluation  Attributes  and  Assessment 

A  completed  matrix  of  the  attributes  and  assessed  values  (as  defined  in  Table  1),  including  the 
rationale  for  each  assessment  and  a  general  description  of  how  the  enabler  could  be  of  value  in 
improving  the  agility/adaptability/responsiveness  of  systems  engineering,  and  the  rationale  for  the 
decision 

Part  III:  Recommendation  Details 

If  the  recommendation  is  for  further  research,  then  one  or  two  specific  studies/experiments/analyses 
that  would  lead  to  the  enabler's  validation  or  support  its  transition  should  be  described.  If  the 
recommendation  is  for  expedited  transition,  a  description  of  why  the  team  believes  this  is  possible, 
what  type  of  transition  materials  exist  or  need  to  be  created,  and  identification  of  organizations  that 
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would  be  appropriate  as  pilots.  If  the  enabler  is  deemed  not  suitable,  no  further  information  is 
required. 

Part  IV:  Previous  Research 

Previous  research  and  experience  in  the  area  of  interest  that  supports  this  possible  usefulness. 

Part  V:  References 
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